Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for WP Rest API paging #1733

Merged
merged 15 commits into from
Aug 10, 2017
Merged

Support for WP Rest API paging #1733

merged 15 commits into from
Aug 10, 2017

Conversation

alexbassy
Copy link
Contributor

Package: gatsby-source-wordpress

Currently you have to increase the default per_page value of the REST API in functions.php to get more posts. In this PR, the X-WP-Total header is used to create multiple async requests* for each WP entity type in order to get all of the entities without having to change anything in Wordpress

Before:
Max 10 posts returned without editing Wordpress backend

Now:
All entities retrieved with no Wordpress modification

image

* Waterfall approach might be better in terms of server load, but can take a lot longer if there are a lot of pages. Perhaps adding request-rate-limiter (or equivalent) might help with this

P.S. Sorry about the diff, my editor removes trailing whitespace

@gatsbybot
Copy link
Collaborator

gatsbybot commented Aug 6, 2017

Deploy preview ready!

Built with commit f10625f

https://deploy-preview-1733--gatsbygram.netlify.com

@gatsbybot
Copy link
Collaborator

gatsbybot commented Aug 6, 2017

Deploy preview ready!

Built with commit f10625f

https://deploy-preview-1733--using-drupal.netlify.com

@KyleAMathews
Copy link
Contributor

Sweet! Thanks for helping out!

To clean up the diff, run "npm run format" from the base of the repo.

@KyleAMathews
Copy link
Contributor

@sebastienfi could you handle reviewing and merging this PR?

@sebastienfi
Copy link
Contributor

sebastienfi commented Aug 7, 2017

Awesome!

@alexbassy

  • Could you please update your branch to the latest ? Example on SOF
  • Don't need to pass perPage as a parameter because it's already "global" (_perPage)
  • Should we keep the default perPage value to 10 or increase it right away to 100 ?
  • Can you update the gatbsy-config.js of the example site using-wordpress and test it with the actual configuration ? The request will run against wordpress.com and you will see a lot of errors happening there. You may find this usefull to developp with the example site:
bash #1 : cd gatsby
bash #1 : npm run watch
... and keep it running

bash #2 : cd gatsby/example/using-wordpress
bash #2 : gatsby-dev
... and keep it running

bash #3 : cd gatsby/example/using-wordpress
bash #3 : gatsby-dev --set-path-to-repo ../your-path-to-gatsby-root/
bash #3 : npm i && npm run dev

@cr101
Copy link

cr101 commented Aug 7, 2017

Will this work for custom post types as well?

@sebastienfi
Copy link
Contributor

@cr101 As of yet, @alexbassy 's implementation will give the per_page querystring parameter to each eandpoint, so it should work. Would you like to clone the topic branch from @alexbassy 's repo and give it a go ?

@cr101
Copy link

cr101 commented Aug 8, 2017

@sebastienfi Sorry, I'm currently unable to take this for a spin as I'm recovering from a nasty head cold

@alexbassy
Copy link
Contributor Author

@sebastienfi thanks for the instructions, I found them very useful. I ran the example using-wordpress my version of the source plugin and it worked fine. The WP example only has 4 posts, so to test the "perPage" param, I set the value to 1, and it requested 4 pages as expected.

  • Rebased
  • Set default perPage to 100
  • Stop passing around perPage
  • Use x-wp-totalpages header rather than calculating

@sebastienfi
Copy link
Contributor

sebastienfi commented Aug 9, 2017

👍

One more thing

@alexbassy
Copy link
Contributor Author

@cr101 yes! I just tested it using the "Custom Post Type UI" plugin with these options:

image

and it worked just fine, the query was allWordpressWpRecipes 👍

@sebastienfi
Copy link
Contributor

@KyleAMathews that's all good for me. I can't merge myself, would you manage it ?

image

@@ -8,6 +8,13 @@
"lodash": "^4.17.4",
"qs": "^6.4.0"
},
"devDependencies": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove all the babel stuff here? We maintain a common babel setup at the base of the repo. If it's not adequate then please suggest/make modifications there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, done. How do you run npm run watch using the config from the base of the repo?

@KyleAMathews
Copy link
Contributor

@sebastienfi hmmm lemme look into this.

@KyleAMathews
Copy link
Contributor

@sebastienfi you should have merge privileges now.

PLEASE always "Squash and merge". Makes the history much more readable.

screen shot 2017-08-10 at 9 41 30 am

@KyleAMathews KyleAMathews merged commit f9b9f74 into gatsbyjs:master Aug 10, 2017
@KyleAMathews
Copy link
Contributor

Thanks @alexbassy! Making a new release now.

@alexbassy
Copy link
Contributor Author

Sweet thanks @KyleAMathews. Is there any list of wanted features for this plugin? I might be working with it a fair bit now, so I'd like to contribute back to the project where I can. One of the things I've done was adding posts to categories as child nodes.

@KyleAMathews
Copy link
Contributor

@sebastienfi has been driving this and I know from private conversations has a TODO list for some sites he's working on. You want to make an issue for that Sébastien?

@alexbassy alexbassy deleted the topics/wordpress-paging branch August 16, 2018 10:40
@jlengstorf
Copy link
Contributor

Hiya @alexbassy! 👋

This is definitely late, but on behalf of the entire Gatsby community, I wanted to say thank you for being here.

Gatsby is built by awesome people like you. Let us say “thanks” in two ways:

  1. We’d like to send you some Gatsby swag. As a token of our appreciation, you can go to the Gatsby Swag Store and log in with your GitHub account to get a coupon code good for one free piece of swag. (We’ve got t-shirts and hats, plus some socks that are really razzing our berries right now.)
  2. If you’re not already part of it, we just invited you to join the Gatsby organization on GitHub. This will add you to our team of maintainers. You’ll receive an email shortly asking you to confirm. By joining the team, you’ll be able to label issues, review pull requests, and merge approved pull requests.

If you have questions, please don’t hesitate to reach out to us: tweet at @gatsbyjs and we’ll come a-runnin’.

Thanks again! 💪💜

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants